A Model for Segment-Based Speech Recognition

Currently, most approaches to speech recognition are frame-based in that they represent the speech signal using a temporal sequence of frame-based features, such as Mel-cepstral vectors. Frame-based approaches take advantage of efficient search algorithms that largely contribute to their success. However, they cannot easily incorporate segment-based modeling strategies that can further improve recognition performance. For example, duration is a segment-based feature that is useful but difficult to model in a frame-based approach. In contrast, segment-based approaches represent the speech signal using a graph of segment-based features, such as average Mel-cepstral vectors over hypothesized phone segments. Segment-based approaches enable the use of segment-based modeling strategies. However, they introduce multiple difficulties in recognition that have limited their success. In this work, we have developed a framework for speech recognition that overcomes many of the difficulties of a segment-based approach. We have published experiments in phone recognition on the core test set of the TIMIT corpus over 39 classes [1]. We have also run preliminary experiments in word recognition on the December '94 test set of the ATIS corpus. In our segment-based approach, we hypothesize segments prior to recognition. Previously, our segmentation algorithm was based on local acoustic change. However, segmentation depends on contextual factors that are difficult to capture in a simple measure. We have developed a probabilistic segmentation algorithm called " segmenta-tion by recognition " that hypothesizes segments in the process of recognition. Segmentation by recognition applies all of the constraints used in recognition towards segmentation. As a result, it hypothesizes more accurate segments. In addition, it adapts to all types of variability, focuses modeling on confusable segments, hypothesizes all types of units, and uses scores that can be re-used in recognition. We have implemented this segmentation algorithm using a backwards A* search and a diphone context-dependent frame-based phone recognizer. In published TIMIT experiments , we have reported an 11.3% reduction in phone recognition error rate from 38.7% with our previous acoustic segmenta-tion to 34.3% with segmentation by recognition [1]. In segment-based recognition, the speech signal is represented using a graph of features. Probabilistically, it is necessary to account for all of the features in the graph. However, each path through the graph directly accounts for only a subset of all features. Previously, we modeled the features that are not in a path using a single " anti-phone " model [2]. However, the features that are not in a path depend …